Sync mad-rccl with develop - #245

…ement (ROCm#206)

* vllm_dissag: GLM-5.1-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement

Adds GLM-5.1-FP8 (GlmMoeDsaForCausalLM = MLA + DeepSeek Sparse Attention) to the
MoRI-EP WideEP disaggregated serving path, stacked on the ROCm#171 unified launcher.
Fully isolated from DeepSeek-V3/R1: GLM gets its own image + a MODEL_NAME-gated
runtime path, so existing models are byte-identical to develop.

Defects fixed (validated 1P/1D EP8 + 2P/2D EP16, NIAH 2k-35k = 10/10, no crash):
- Long-context accuracy collapse: vLLM #47766 cache-key fix keeps the persistent
  sparse-MLA kernel ON (keys metadata on per-request context+query len).
- 8k disagg prefill crash: DSA adds a 2nd (indexer) KV cache per layer that the
  single-geometry MoRIIO connector never transferred; paired + shipped prefill->
  decode. Plus DSA invalid-token kernel fix (#45324) and shik-latest DP-notify.

Changes:
- docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile: NEW per-model image
  (raviguptaamd/vllm glm5.1-dsa-wideEP_on_shik_latest + aiter e03fa6040 + mori
  42e895472b08 + router). The base vllm_disagg_inference Dockerfile (DSV3/R1) is
  left untouched. Future models add their own Dockerfile the same way.
- models.json: card pyt_vllm_disagg_mori_glm-5.1-fp8 (GLM_SKIP_PATCHERS=1: image
  carries the DSA fixes in-source).
- models.yaml: GLM-5.1-FP8 recipe (block=1, AITER MLA on, eager, mori backends).
  DeepSeek-V3 dp: caps (--max-num-seqs 64 --max-model-len 32768) to bound the
  newer base's decode logits workspace (isolated to the DSV3 entry).
- connectors/moriio.sh: MODEL_NAME-gated GLM DSA runtime patchers (pure no-op for
  other models); GLM_SKIP_PATCHERS switch for baked-fix images.
- 9 idempotent, anchor-based, self-skipping GLM DSA patcher scripts.

KNOWN OPEN DEFECT (future work): 4P/4D EP32 emits corrupted tokens at all context
lengths (suspect moriep all-to-all combine at scale); use 1P/1D and 2P/2D.

Co-Authored-By: Claude <noreply@anthropic.com>

* vllm_dissag: make NIAH harness thinking-model-aware (GLM-5.1)

benchmark_niah.py mis-scored thinking models: it never disabled thinking and
read only content + reasoning_content. GLM-5.1 emits chain-of-thought into the
`reasoning` field and leaves `content` empty until the final answer, so with a
small max_tokens the answer never lands in content -> a false 0/10 even when
generation is correct.

- Add chat_template_kwargs.enable_thinking=false so the answer goes to content.
- Also score the `reasoning` field as a fallback.

Verified against GLM-5.1-FP8: correct 9-10/10 retrieval across 2k-35k on all
tested topologies (EP8/EP16/EP32) after the fix.

Co-Authored-By: Claude <noreply@anthropic.com>

* vllm_dissag: NIAH multi-seed support (NIAH_SEEDS) for variance-aware accuracy

Needle layout is seeded, so a single run is deterministic (bit-exact on the same
stack) but can't tell a real accuracy dip from single-needle variance. Add
NIAH_SEEDS (default 0,1,2) to run each context length across multiple needle
layouts; the summary now reports mean/min/max across seeds. Backward compatible:
NIAH_SEEDS=0 reproduces the prior single-seed behavior.

Co-Authored-By: Claude <noreply@anthropic.com>

* vllm_dissag: NIAH gate robust to cold-start JIT (warmup + readiness probe)

On a freshly-booted node the first request of each context shape pays the full
JIT/kernel-autotune compile (minutes). The NIAH harness scored the FIRST request,
so cold compile landed on a scored/gated request -> false 0/10 or timeout, failing
the accuracy gate and skipping the perf sweep. Root-caused by reproducing on a cold
boot (0 results) vs a warm server (all pass) on the same image.

Fixes:
- benchmark_niah.py: add a warmup pass (NIAH_WARMUP=1 default) — one throwaway
  request per context length before scoring, with a generous timeout, failures
  tolerated. Scored requests are then always warm.
- benchmark_niah.py: distinguish TIMEOUT/ERROR from a wrong answer. Timeouts return
  a sentinel (excluded from mean, never counted as 0/10); summary flags NO-RESULT
  with guidance instead of silently reporting 0.
- benchmark_niah.sh: replace the blind `sleep 10` with a /v1/models readiness poll
  (up to 5 min), and forward NIAH_WARMUP.

Verified: patched harness on the warm server passes 10/10; cold-boot repro no longer
produces false 0/10 because compile happens in the warmup pass.

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] v0.27 level-set: 3.4x decode speedup, 200K context, RDMA + launcher fixes

Validated on MI300X, 8 nodes, image rocmshared/pytorch-private:glm5.1-vllm027-b8
(base ci_base-dedbf6be8b + vLLM raviguptaamd/vllm@glm5.1-dsa-wideEP_on_vllm-v0.27
+ aiter e03fa6040 + MoRI 42e895472b08 + router ROCm#181).

PERF FIX (models.yaml decode.dp) -- the headline change:
  --max-num-batched-tokens 2048 on the DECODE role only.
  max_num_batched_tokens is a chunked-prefill SCHEDULER knob, but it also sizes the
  MoRI EP dispatch buffer (fused_moe/layer.py -> all2all_utils.py -> all2all.py
  max_num_inp_token_per_rank). At the 8192 default a decode instance ran an
  8192-token-wide all2all every step, per layer, x78 layers, while decoding a handful
  of tokens: a fixed ~302ms/step floor, ~320x this model's HBM-bandwidth bound.
  Prefill keeps 8192 (it genuinely dispatches wide batches).

  1024/64 con=8, warm:      TPOT        TTFT        out tok/s
    1P/1D  302 ->  88.0 ms   2431 -> 906 ms   24.9 -> 78.8
    2P/2D  302 ->  94.1 ms          1633 ms          66.7
  Published reference: 1P/1D ~89ms, 2P/2D ~91ms -> matched within 3%.
  Accuracy unaffected: NIAH 2k-200k clean on both topologies, no length collapse,
  memfault=0, latencies equal-or-better at every length. 200K validated (5.7x beyond
  the previously published 35K ceiling).

Dockerfile: base -> ci_base-dedbf6be8b (matches the fork's upstream base), VLLM_REF ->
  the v0.27 branch, and WITH_MORI_BUILD/WITH_AITER_BUILD now default to 1 so a plain
  `docker build` reproduces the validated stack. Previously they defaulted to 0, which
  silently used the base's bundled aiter 0.1.19 -- that GPU-faults on the GLM DSA decode
  kernel. The pinned aiter e03fa6040 / MoRI 42e895472b08 must not be bumped without
  re-running long-context NIAH.

connectors/moriio.sh: per-role env split (PREFILL_*/DECODE_* -> VLLM_MORI_*), mirroring
  the existing PREFILL/DECODE_MORI_BACKEND pattern -- models.yaml env: applies to BOTH
  roles, but prefill and decode need opposite values here. Also injects
  use_inductor_graph_partition (pairs with the vLLM splitting_ops fix).

connectors/moriio.env: RDMA fabric -- MORI_IB_GID_INDEX=3 (RoCEv2 IPv4),
  MORI_RDMA_DEVICES/NCCL_IB_HCA restricted to the 8 GPU-local NICs (leaving the mgmt
  NICs in makes QPs form over a non-routable fabric -> ibverbs.cpp:189 timeouts),
  NCCL/GLOO control sockets on eth0.

run_xPyD_models.slurm: libionic bind-mount requires a regular file after symlink resolve
  (a dangling symlink gave "OCI runtime create ... not a directory", container exit 125);
  prefer FABRIC_SUBNET over `hostname -I` first IP (nodes list a 10.224 overlay first,
  which made the socket_barrier advertise an unreachable NIC -> "Waiting for nodes" hang);
  GLM_KERNEL_PATCH/GLM_BACKEND_PATCH bind-mount hooks to test .py fixes without a rebuild;
  forward the new per-role env keys.

vllm_disagg.sh: same FABRIC_SUBNET IP-selection fix for host_ip.

benchmark_xPyD.sh: per-shape warmup at the REAL isl/osl before each shape's cells. The
  global warmup is isl=osl=32/con=1, which never exercises a 1024/8192/28672 prefill path
  or the decode cudagraph batch sizes, so the first measured cell absorbed residual JIT
  (observed 302ms vs ~88ms steady-state). Warmup output goes to a separate
  _SHAPEWARMUP.log so it cannot pollute the CSV.

models.yaml (GLM-5.1-FP8): decode.dp perf fix above; recipe = prefill eager +
  mori_high_throughput, decode PIECEWISE cudagraph + mori_low_latency; VLLM_USE_LAYERNAME=0;
  VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=64; NCCL heartbeat/timeout knobs for long-context
  collectives.

Full operational playbook (including the dead ends) in skills_vllm_disagg.md.

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Default WITH_NIXL=0 (lean MoRI-EP-only image)

GLM-5.1 is served over MoRI-EP (all2all) + MoRI-IO (KV transfer); the rixl connector's
UCX/RIXL/rocSHMEM/DeepEP stack is never selected by this recipe, so building it only
lengthens the build and ships unused transports.

0 is also how the validated image was actually built: glm5.1-vllm027-b8 used
--build-arg WITH_NIXL=0, so the previous default of 1 meant a plain `docker build`
did NOT reproduce the tested artifact. With this change, and with
WITH_MORI_BUILD/WITH_AITER_BUILD already defaulting to 1, a no-flag build now matches
the validated stack exactly.

Set --build-arg WITH_NIXL=1 if you need the rixl connector from this same Dockerfile.

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Add long-context NIAH harness + vllm-disagg operational playbook

niah_200k.py: needle-in-a-haystack sweep that validated GLM-5.1-FP8 to 200,049 tokens
on both 1P/1D (EP8) and 2P/2D (EP16). Reports found/10, latency, and the server-reported
prompt_tokens per length, and writes JSON. Model id is overridable via NIAH_MODEL so it
is not GLM-specific. The existing benchmark_niah.* stop well short of this range; this
covers the 64k-200k band.

skills_vllm_disagg.md: operational playbook for vLLM PD-disaggregated WideEP on MI300X
(MoRI-EP + MoRI-IO), written from this enablement. Documents, with measurements:
  - benchmarking method: ALWAYS discard the first post-boot run (cold Triton JIT made
    TTFT read 13.4s vs 906ms warm; with prefill eager the JIT cost lands in TTFT, not
    TPOT), and sanity-check against the HBM-bandwidth bound before blaming a kernel
  - the max_num_batched_tokens trap: a chunked-prefill SCHEDULER knob also sizes the
    MoRI EP dispatch buffer, so decode ran an 8192-token-wide all2all every step
    (302ms -> 88ms TPOT once sized for the real batch)
  - the DSA sentinel landmine: the invalid sparse-index sentinel must be 0, not -1,
    because aiter's mla_decode_fwd dereferences it (only bites at disagg long context)
  - three documented DEAD ENDS so they are not retried, including why mori's
    max_total_recv_tokens cannot decouple recv from send capacity (the clamp is a min())
  - cache/boot behaviour (three caches with different rules, the aiter baton lock,
    measured boot times), readiness signals for multi-node topologies, per-role env
    plumbing, and RDMA fabric verification (a node can be SLURM-'alloc' with a dead
    fabric - verify with ping/ib_write_bw before blaming code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Remove dead-end MoRI EP env plumbing and its misleading comments

Review cleanup. The VLLM_MORI_MAX_TOKENS_PER_RANK / VLLM_MORI_MAX_TOTAL_RECV_TOKENS
per-role plumbing was written while chasing the decode-TPOT bug and is NOT what fixed it
(the fix is `--max-num-batched-tokens 2048` on decode.dp). Worse, the comments asserted
that max_total_recv_tokens keeps recv capacity large enough for vLLM's profiling dummy
run -- which is false and was disproved by measurement: mori's

  MaxNumTokensToRecvPerRank() = min(ceil(maxTotalRecvTokens / worldSize),
                                    maxNumInpTokenPerRank)

is a min(), so maxTotalRecvTokens can only LOWER recv capacity, never raise it above the
send width. Anyone following those comments and setting the knobs would hit
"Total recv token overflow" at boot (observed at 512, 2048, and with recv=65536).

Removed: the per-role export block in moriio.sh, the stale models.yaml comment block, and
the six dead keys from _RECIPE_ENV_KEYS. Replaced with a short NOTE in moriio.sh pointing
at the real fix and at skills_vllm_disagg.md for the measurements and dead ends.

No functional change to the validated configuration: the knobs defaulted to 0/unset, so
the b8 runs never exercised them.

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Decode CUDA-graph FULL_AND_PIECEWISE: 2.1x (EP8) / 1.7x (EP16) faster decode

Second perf lever, on top of the decode --max-num-batched-tokens fix. Switching the decode
role from PIECEWISE to FULL_AND_PIECEWISE captures a FULL graph for the uniform-decode
shapes and falls back to PIECEWISE for the rest.

Measured on MI300X, ISL/OSL 1024/64, warm (first post-boot run discarded), image
rocmshared/pytorch-private:glm5.1-pr176-verify:

                        PIECEWISE   FULL_AND_PIECEWISE    gain
  1P/1D EP8  TPOT c8      88.0 ms        41.8 ms         2.11x
  1P/1D EP8  TPOT c16     91.5 ms        45.8 ms         2.00x
  2P/2D EP16 TPOT c8      94.1 ms        55.4 ms         1.70x
  2P/2D EP16 TPOT c16     96.2 ms        60.3 ms         1.60x
  1P/1D outTP c8        78.8 tok/s    143.7 tok/s        1.82x
  2P/2D outTP c8        66.7 tok/s    113.0 tok/s        1.69x
  2P/2D TTFT c16          2221 ms        1400 ms         1.59x
Both topologies now sit well below the published ~89 ms (EP8) / ~91 ms (EP16) reference.

Accuracy is unaffected. NIAH 2k-35k: 1P/1D 51/60, 2P/2D 55/60 - both inside the PIECEWISE
band (52-53 and 53 respectively; 2P/2D is in fact the best NIAH run of the campaign). No
length collapse, no '!!!', memfault=0, latencies equal-or-better at every length.

Caveats, stated rather than buried:
 - The gain SHRINKS as EP widens (2.11x at EP8 -> 1.70x at EP16). FULL captures the whole
   decode step including the cross-node all2all, which does not compress. This is validated
   at EP8 and EP16 ONLY; do not assume it holds at EP32 (which has a separate, unrelated
   correctness regression).
 - Capture costs ~92-94 s and ~3.0-3.5 GiB vs ~5 s / 7.2 GiB for PIECEWISE (fewer but
   larger graphs), so boot is longer. Irrelevant for a long-lived server, notable for CI.
 - Set DECODE_CUDAGRAPH_MODE=PIECEWISE to fall back.

Two code paths branch on CUDAGraphMode.FULL (sparse_attn_indexer.py:411 and the MoRIIO
READ-mode barrier); both are guards that skip host-side work under FULL, so enabling it is
safe here - the MoRIIO one is already a no-op in WRITE mode.

skills_vllm_disagg.md gains a section 2b with the table, the EP-width caveat, the capture
cost, and the observation that FULL is far less cold-JIT sensitive (its warmup run already
reads steady-state TPOT, where PIECEWISE's first run showed 13.7 s TTFT).

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Merge hygiene: drop lab leftovers, safe patcher defaults.

Rebase onto develop after ROCm#197 so the GLM card lives in
scripts/vllm_dissag/models.json. Remove the NFS silly-rename and
keepalive/NIAH-200k/skills lab files, revert DeepSeek-V3 context caps
and cluster-wide moriio.env NIC pins, turn persist-gate and DSA
sentinel patchers opt-in (they crash the v0.27 image), gate DSA on
GLM-5.*, and wire BENCHMARK_SCRIPT=niah for the 1P/1D smoke.

* [GLM-5.1] Point the recipe at the v0.27 image it was validated on.

The recipe named a pre-v0.27 image as "validated" and described patchers
that no longer apply. Serving GLM-5.1-FP8 on that older image returns
empty completions (NIAH 2k = 0/10), so name the image the Dockerfile
actually builds and state that the patchers are an old-image fallback.

* [GLM-5.1] Minimal MAD: drop runtime DSA patchers, make the image pin the contract

PR 206 shipped GLM-5.1-FP8 with nine apply_glm_*.py runtime monkeypatch scripts
(~1100 lines) invoked from the moriio connector. None of them run on the
production path: the validated model card sets GLM_SKIP_PATCHERS=1 because the
pinned vLLM (VLLM_REF=glm5.1-dsa-wideEP_on_vllm-v0.27) already carries every DSA
fix in-source. They were dead weight that MAD had to keep in sync with a fork.

This branch makes MAD carry catalog content only, and makes the image solely
responsible for vLLM source fixes:

- Delete all nine scripts/vllm_dissag/apply_glm_*.py.
- connector_runtime_patch() in connectors/moriio.sh is now a documented no-op for
  every model; _glm_dsa_runtime_patch() and the GLM MODEL_NAME branch are gone.
  Non-GLM behavior is unchanged (it already returned 0 before reaching any GLM
  code), so DeepSeek/Llama/gpt-oss are untouched.
- Remove the plumbing that only existed to feed the patchers: the
  GLM_SKIP_PATCHERS / GLM_PERSIST_GATE / GLM_DSA_SENTINEL_FIX docker -e forwards
  in run_xPyD_models.slurm, and GLM_SKIP_PATCHERS from the model card in
  models.json (with no patchers, "skip the patchers" is meaningless).
- Rewrite the models.yaml and Dockerfile comments that promised runtime patchers.
  The docs now state the actual contract: the image must carry the DSA fixes
  in-source and VLLM_REF is the requirement, not a preference.

Trade-off: this removes the fallback path for older images. On an image built
from a vLLM ref that predates the DSA fixes, GLM-5.1 now boots and then produces
garbage or stalls the disagg KV transfer, with nothing to recover it -- the fix
is to rebuild the image, not to patch at runtime. In exchange MAD stops carrying
a duplicate of upstream fork code that no supported configuration executes.
The Dockerfile, model card, models.yaml recipe, NIAH harness and slurm allowlist
are otherwise unchanged.

Verified without GPUs: tests/argv_assert.sh 25 passed / 0 failed; models.json
parses and the GLM card is intact; DRY_RUN argv for DeepSeek-V3,
DeepSeek-V3-5layer, DeepSeek-R1, amd-Llama-3.3-70B-Instruct-FP8-KV and
gpt-oss-120b is byte-identical to glm5.1-wideep-mad (so the only delta vs
origin/develop is the pre-existing one from that branch); GLM-5.1-FP8 still
emits --enable-expert-parallel, --block-size 1, and on decode
--max-num-batched-tokens 2048 + FULL_AND_PIECEWISE; bash -n clean.

* [GLM-5.1] Scope USE_INDUCTOR_GRAPH_PARTITION to GLM, restore other models' argv

The knob was introduced by the GLM v0.27 level-set (it does not exist on
origin/develop) with a default of 1, so every model served over the moriio
connector silently got "use_inductor_graph_partition":true appended to its
--compilation-config: DeepSeek-V3, DeepSeek-V3-5layer, DeepSeek-R1,
amd-Llama-3.3-70B-Instruct-FP8-KV and gpt-oss-120b. The flag is a fix for the
MLA custom-op boxing failure that only GLM-5.1 hits, so it belongs with the
model, not with the connector.

Flip the connector default to 0 and set USE_INDUCTOR_GRAPH_PARTITION=1 in the
GLM-5.1-FP8 models.yaml env: block, the same mechanism the recipe already uses
for VLLM_USE_LAYERNAME / KV_BLOCK_SIZE / DECODE_CUDAGRAPH_MODE. Add the key to
_RECIPE_ENV_KEYS so the image-baked < models.yaml < submit-time -e precedence
holds for it too (without it a submit-time override would lose to the recipe).

Verified offline (DRY_RUN argv, no GPUs): the five models above are now
byte-identical to origin/develop across moriio+rixl x wideEP+TP x
prefill+decode (40 cells, zero diff, no use_inductor_graph_partition anywhere),
GLM-5.1-FP8 prefill+decode argv is unchanged vs glm5.1-wideep-mad, and
argv_assert is 25 passed / 0 failed.

* [GLM-5.1] Fix false MORI_REF claim and stale image text in the GLM docs

Comment-only. Three claims in the GLM catalog text did not match reality:

1. connectors/moriio.sh said MORI_REF=42e895472b08 had the large-transfer
   notify/mapping fixes #424/#436/#432 "baked in". It does not: the pin is
   ROCm/mori#366 (2026-06-05), while #424 merged 2026-06-26 and #432/#436 on
   2026-06-29 — `gh api repos/ROCm/mori/compare/42e895472b08...<merge_sha>`
   reports status "ahead" for all three. The pin is left alone (the image was
   validated with exactly this MoRI); only the comment is corrected.

2. models.yaml said the unsupported rocmshared lab image "is only usable on a
   branch that still ships patchers" — a dangling reference from a branch with
   no patchers. It now names the one supported image outright.

3. models.yaml's long-context caveat described the old mori v1.2.1 lab image
   ("garbage beyond ~16-18k tokens") as if it applied to the pinned image, which
   contradicts both the glmv5.1 Dockerfile STATUS and the NIAH 2k-35k numbers in
   this same recipe. Re-scoped to history and pointed at the in-source fix
   (vLLM #47766). #40018 is also labelled as an issue, which it is.

Two audited citations needed no change: vLLM #47042 is only cited in
benchmark_niah.{py,sh} / README.MD / ARCHITECTURE.md, all of which are
origin/develop text that already calls it an issue (never a PR); and aiter#3921
(a gfx950 A8W8 change) was only cited by apply_glm_dsa_persistent_kernel_gate_fix.py,
which this branch already deleted.

* [GLM-5.1] Fix cross-model fallout: protect-list, fabric-IP default, patch mounts

Three defects that reached beyond GLM-5.1-FP8:

1. _RECIPE_ENV_KEYS listed only 18 of the 39 keys a models.yaml `env:` block can
   set. Once MODELS_YAML_PROTECT exists the driver protects ONLY listed keys, so a
   submit-time `-e KEY=...` silently lost to the recipe for the 14 unlisted non-GLM
   keys (Llama-405B/70B, DeepSeek family, gpt-oss-120b) and 7 GLM ones. That
   contradicts the precedence documented here, in models.yaml and in
   ARCHITECTURE.md. The list is now the exact union of models.yaml env: keys.

2. FABRIC_SUBNET defaulted to the lab prefix 10.158., changing which NIC every
   model advertises on any cluster that happens to have a 10.158.x address, and the
   slurm never forwarded it, so an operator override applied on the host but not
   inside the container (split-brain MASTER_ADDR vs host_ip). Now opt-in: unset =
   hostname -I's first entry, i.e. develop's behaviour, and the var is forwarded.

3. GLM_KERNEL_PATCH / GLM_BACKEND_PATCH bind-mounts overlaid vLLM sparse-MLA source
   from a host path. They do not exist on develop, nothing else in the tree reads
   them, and a host-path source overlay is precisely the runtime patching this
   branch removed in favour of the image pin. Dropped.

Also correct the `-e` -> `-f` rationale on the RDMA lib mounts: a dangling symlink
already fails -e; what -f excludes is a path that resolves to a non-regular file.

* [GLM-5.1] Correct false and stale claims in the GLM docs

Verified against ROCm/mori, vllm-project/vllm and the pinned vLLM fork with gh api:

- Dockerfile stage 1 claimed the MoRI pin is "v1.2.1 (the version for the 06_29
  mori121 image)". MORI_REF=42e895472b08 is ROCm/mori#366 (2026-06-05) and is 32
  commits BEHIND tag v1.2.1 (e31d426a, 2026-06-25). The text was copied from the
  base Dockerfile, which really does pin v1.2.1, and it also inverted models.yaml,
  which lists the mori121 image as NOT SUPPORTED.

- Dockerfile stage 3 claimed the vLLM pin is "the 06_29 validated Wide-EP WRITE-mode
  branch (matches the published mori121 image)". It is the GLM DSA v0.27 branch; the
  06_29 vLLM predates the in-source DSA fixes and cannot serve GLM-5.1-FP8 at all.

- VLLM_REF is documented as "(HEAD cda3648602) = upstream v0.27 tip + 7 commits".
  The branch tip is d723eb305 (+9); it advanced on 2026-08-15, before the MAD commit
  that measured the 3.4x decode speedup. Since VLLM_REF is a branch name, `docker
  build` resolves it to that day's tip. Record the real state and tell builders to
  pass an explicit --build-arg VLLM_REF=<sha> for an auditable rebuild.

- STATUS said "NIAH 2k-35k = 10/10" where the recipe records 51/60 and 55/60, and
  the vLLM pin note said "1P/1D + 2P/1D + 1P/2D ... decode PIECEWISE" where the
  validated topologies are 1P/1D EP8 + 2P/2D EP16 and decode is FULL_AND_PIECEWISE.

- The `docker build -f` example inside the glmv5.1 Dockerfile named the BASE
  Dockerfile.

- models.yaml's "3 recipe-defining ways" bullet still described decode as PIECEWISE
  with a ~3.7x ITL win after 638a033 moved it to FULL_AND_PIECEWISE. Reframed: the
  ~3.7x is PIECEWISE over eager, FULL_AND_PIECEWISE is the further 2.11x/1.70x
  already measured below.

- benchmark_xPyD.sh attributed "302ms vs ~89ms" to residual JIT absorbed by the
  first measured cell. models.yaml and moriio.sh attribute the same 302->88ms to
  the 8192-token-wide MoRI EP all2all buffer, fixed by --max-num-batched-tokens.
  Note the distinction rather than leaving two root causes for one number.

- vLLM #40018 is filed against gfx950 while this stack is gfx942; say so.

- README told users to build "the one fullsource Dockerfile" and never mentioned
  that GLM-5.1-FP8 needs the per-model glmv5.1 image the same README's table cites.

- benchmark_niah.sh passes every NIAH_* through explicitly except NIAH_SEEDS, which
  only worked by inheritance.

* [GLM-5.1] Sync the gate_check allowlist mirror with the slurm

The branch added GLM-5.1-FP8 to VALID_MODELS, MORI_EP_VALID_MODELS and
WIDE_EP_ONLY_MODELS in run_xPyD_models.slurm but not to gate_check.sh's copy, so
the new model's gate had no coverage. Add it, plus the four cases that pin the
intended envelope (moriio wideEP only; TP and DeepEP rejected).

The header also claimed the test "runs the REAL gate code path, so a future edit to
the lists/gate is caught" — it does not, it re-runs a hand-kept mirror, which is
exactly why this drift was possible. Say what it actually does.

* [GLM-5.1] Name the validated v0.27 image tag and the 0/10 evidence.

The IMAGE CONTRACT block said which ref the supported image is built from but
never named the published artifact, and gave no evidence for why older images
are unsupported. Name the tag (rocmshared/pytorch-private:glm5.1-vllm027-b8)
and record the measurement: the pre-v0.27 image
rocm/pytorch-private:glm-dockerimage-built-09072026 scored NIAH 2k 0/10 on the
1P/1D EP8 smoke, slurm job 216847.

This branch ships no runtime patchers, so an unsupported image has no fallback
and raises no error -- it serves and returns wrong output. Say that plainly.
The glmv5.1 Dockerfile gets the same tag and the same evidence so the two files
name one artifact; its vLLM stage already states that the mori121 image's vLLM
cannot serve GLM-5.1-FP8, which is what models.yaml says too.

Comment-only: no flags, no env, no pins move.

* [GLM-5.1] Pin VLLM_REF to the validated commit instead of a branch name

VLLM_REF defaulted to glm5.1-dsa-wideEP_on_vllm-v0.27, a mutable branch name, so
`docker build` resolved to whatever the tip was on the day of the build. The branch
has already moved once since validation (cda3648602 -> e8c186f71/d723eb305), which
means "the image is the contract" could not hold: two builds of the same Dockerfile
could ship different engines.

Pin the tip that every number in models.yaml was measured on, d723eb305e, and update
the places that described the ref as a branch (models.yaml, connectors/moriio.sh, the
Dockerfile header). Moving the pin now requires an explicit --build-arg and a
re-validation, which is the intent.

* [GLM-5.1] Correct the described lineage of the pinned vLLM commit

The header called dedbf6be8b the "upstream v0.27 tip". It is not: against
vllm-project/vllm, dedbf6be8b is a main-branch commit of 2026-08-09, 270 commits
ahead of the v0.27.0 tag and 17 behind releases/v0.27.0. Verified with the GitHub
compare API, which also confirms the pinned d723eb305e is that commit plus exactly
9 ROCm commits, 0 behind.

State what the commit actually is so a reader does not go looking for these fixes on
the v0.27 release branch.

* [GLM-5.1] Point VLLM_REF back at the branch, pending the mutable-ref fix

Reverts the SHA pin from the previous commit. The ref goes back to
glm5.1-dsa-wideEP_on_vllm-v0.27 so this Dockerfile stays consistent with how the
other vllm_dissag images name their vLLM ref; the mutable-branch problem is being
handled separately rather than solved one Dockerfile at a time.

The REPRODUCIBILITY note stays and now names the validated commit outright, so an
auditable rebuild is still one build arg away:
  --build-arg VLLM_REF=d723eb305eb78d1bda0ed357b2b54cc29487221f

The lineage correction from the previous commit is kept and reworded for the branch
form: the tip is dedbf6be8b plus 9 ROCm commits, and dedbf6be8b is upstream main, not
the v0.27 release.

* [GLM-5.1] Trim the connector_runtime_patch comment back to develop's

connector_runtime_patch is a return-0 no-op shared by every moriio model, and this
branch had grown its comment from 7 lines to 24.

The MoRI paragraph was wrong for the model that dominates this connector: it told the
reader their pin is 42e895472b08 and lacks ROCm/mori #424, #432 and #436, but
DeepSeek-V3 builds from vllm_disagg_inference.ubuntu.amd.Dockerfile with
MORI_REF=v1.2.1, 32 commits ahead of that sha and containing those fixes. The pin
story is already told, correctly scoped to the GLM image, in the glmv5.1 Dockerfile.

The GLM paragraph restated a rule the existing text already makes generic and
duplicated detail that belongs in the Dockerfile header and the models.yaml recipe.

Restores develop's wording plus one line pointing GLM readers at models.yaml, so the
hunk is a one-line diff against develop.

* [GLM-5.1] Drop the GLM line from connector_runtime_patch

The function is unedited by this branch, so it needs no note from it. Leaves
connector_runtime_patch byte-identical to develop; the GLM image contract is stated
in models.yaml and the glmv5.1 Dockerfile, which is where it belongs.

* [GLM-5.1] Correct the necessity claim for use_inductor_graph_partition

The comments in moriio.sh and models.yaml stated the flag was required for GLM,
citing "RuntimeError: unknown parameter type" on the first real MLA decode from the
stable-ABI concat_and_cache_mla boxed kernel failing to compose in the Dynamo-FX
partitioned graph.

That does not reproduce on the pinned image. A/B on 1P/1D EP8 (slurm 217352.91 with
the flag off vs .108 with it on) scored NIAH 2k 10/10 in both arms, with the emitted
--compilation-config verified per arm from the live engine cmdline, and zero
occurrences of that RuntimeError in any log. A matched 1024/1024 concurrency-8 sweep
put mean TPOT at 42.57 ms off vs 42.11 ms on, inside run-to-run noise.

Comments only; no behavior change. GLM keeps the flag ON because that is the
configuration every published number was measured with, which the comments now say
instead of claiming a correctness requirement.

* [GLM-5.1] Drop the models.yaml protect-list, restore develop's env precedence

_RECIPE_ENV_KEYS existed to add a third precedence tier so models.yaml could
override image-baked ENV while a submit-time -e still won. That tier only
matters if a supported image bakes a recipe key, and none does: docker inspect
of the glmv5.1 image shows no KV_BLOCK_SIZE, no VLLM_ROCM_USE_AITER_MLA and no
cudagraph settings -- its only ENV lines are build and toolchain vars. The case
it defended is a lab image baking DeepSeek tuning, already unsupported for GLM.

Against that it required a 35-key list in the slurm to be kept in sync with
every models.yaml env: block, in a different file, or the documented precedence
silently breaks for whichever key was missed.

vllm_disagg.sh's yaml-env block is now byte-identical to develop. Diff versus
develop drops from 79 to 44 lines in the slurm and 38 to 9 in the driver, the
remainder being the unrelated FABRIC_SUBNET fix.

Verified: DRY_RUN argv for GLM-5.1-FP8, DeepSeek-V3 and DeepSeek-R1 identical
before and after; GLM still emits use_inductor_graph_partition. Submit-time
overrides still win (USE_INDUCTOR_GRAPH_PARTITION=0 drops the key,
KV_BLOCK_SIZE=16 beats the recipe's 1) because an exported value is in
os.environ under develop's rule. tests/run_all.sh 25/25 plus the gate suite.

* [GLM-5.1] Make per-shape benchmark warmup opt-in, not default-on

benchmark_xPyD.sh is develop's default sweep path, shared by every model, so
gating per-shape warmup on ${SHAPE_WARMUP:-1} would have changed the measured
TPOT of recipes already validated without it (DeepSeek-V3/-R1, Llama-70B,
gpt-oss-120b). Default to off and let GLM-5.1-FP8 opt in from its models.yaml
env: block, which is the configuration its published latency numbers were
measured under. Other models' sweeps are now byte-identical to develop.

Also forward SHAPE_WARMUP and USE_INDUCTOR_GRAPH_PARTITION into the container
conditionally. docker run only passes variables named in an explicit -e, so
without this a submit-time export of either key never crosses the container
boundary and models.yaml applies unopposed -- meaning neither knob could be
A/B-tested from a wrapper script. Unset means not forwarded, so neither can
shadow a recipe on a normal run.

Covered by five new assertions in tests/argv_assert.sh, including that
GLM-5.1-FP8 is the only warmup opt-in.

* [GLM-5.1] Trim the comment bloat out of the GLM enablement diff

The GLM changes added 409 comment lines against 271 lines of code, a 60%
comment ratio that buried the actual recipe. models.yaml carried 130 comment
lines for one model entry where the DeepSeek entries use roughly 17, and
moriio.sh -- a file every model shares -- carried 25 comment lines for 6 lines
of code. This removes 131 of them without touching a single line of code.

Three kinds of comment are gone. Duplication: the MoRI EP buffer-width
rationale was written out in full in both moriio.sh and models.yaml, so it now
lives only at decode.dp where the setting actually is, and the same applies to
the inductor-partition and FABRIC_SUBNET explanations. Lab narrative: the
long-context collapse history, the unsupported-image inventory and the branch
archaeology described a superseded image rather than the shipped one. Internal
evidence: slurm step IDs, build labels and GitHub compare arithmetic that no
reader outside this lab can act on.

What stays is anything a future editor would break by removing the code it sits
on -- KV_BLOCK_SIZE=1 for the DSA indexer, the prefill cudagraph deadlock, the
sparse-indexer crash fix, the decode.dp TTFT floor -- plus the two safety
warnings that describe SILENT failures: unsupported images return confident
garbage rather than erroring, and long-context was fixed in-source so it must
not be chased with config flags. Measurements are kept only as one-liners where
they justify a non-obvious setting; the full tables moved to the work log.

The Dockerfile is trimmed against its sibling vllm_disagg_inference Dockerfile
rather than an arbitrary target: blocks copied from it are left byte-identical,
and only the GLM-specific additions are compressed.

Comment-only, proven three ways: the non-comment diff is empty, DRY_RUN argv is
byte-identical for GLM, DeepSeek-V3, DeepSeek-R1, Llama-70B and DSV3+DeepEP,
and the offline suites stay green at 24 gate + 30 argv assertions.

* [GLM-5.1] Correct the per-shape warmup rationale to match measurement

Both comments claimed the first measured cell of a shape absorbs residual JIT
and reports an inflated TPOT. An A/B on allocation 217352 at 1024/1024 con=8
does not support that: with the warmup on TPOT was 42.13 ms and with it off
42.05 ms, and the ~21 s P99 TTFT in the con=1 cell that follows the warmup is
present either way. State the real reason the recipe keeps it -- GLM's published
figures were measured with it on -- rather than a mechanism that was not
observed. Comments only; no code changes.

* [GLM-5.1] Keep the FULL_AND_PIECEWISE measurement table in models.yaml

The previous commit reduced this block to the two TPOT figures on the grounds
that the full table was PR-description material. That was premature: the
throughput column, the speedup multipliers and the NIAH accuracy row are the
baseline the upstream DSA PRs get verified against, and they need to be
readable next to the setting until those PRs merge.

Restores the two measured rows and the accuracy line, keeps the tightened
prose around them, and says in the comment that the rows are a temporary
verification baseline so a later reader knows they can go once upstream lands.

Comment-only: non-comment diff empty, DRY_RUN argv byte-identical for GLM,
DeepSeek-V3, DeepSeek-R1, Llama-70B and DSV3+DeepEP, offline suites green at
24 gate + 30 argv.

* [GLM-5.1] Drop the FABRIC_SUBNET opt-in; it fixes nothing by default

The launcher advertises hostname -I's first entry on both sides of the
container boundary, and socket_barrier binds --local-ip host_ip while peers
dial --node-ips IPADDRS. When that first entry is not routable between nodes
the barrier never completes and the job hangs at "Waiting for nodes" with no
error. That is a real defect on this cluster: job 211823 came up with
IPADDRS=10.224.2.61,10.224.0.194 and hung, and 211879 had one node on the
overlay and one on the fabric and hung with a single engine up. Both predate
GLM, so it is a cross-model launcher defect.

The mitigation carried here was an opt-in FABRIC_SUBNET prefix filter applied
to both lookups. Defaulting to unset, it fixes nothing for anyone: the next
time enumeration flips, the run hangs exactly as before because nobody exports
the variable. It was documented in no README, mentioned in no PR text and
covered by no test, which makes it dead code in an enablement PR. A real fix
means automatic routable-NIC selection, which changes NIC choice for every
model on every cluster and belongs in its own launcher PR with its own testing.

Reverting also removes a latent defect: _pick_fabric_ip had dropped develop's
NR==1 guard, so a multi-line srun stdout would have yielded a multi-line
MASTER_ADDR.

vllm_disagg.sh is now byte-identical to develop, so this PR no longer touches
it, and the slurm diff drops from 46 lines to 21. DRY_RUN argv is unchanged for
GLM-5.1-FP8, DeepSeek-V3 and Llama-70B; suites stay green at 24 gate + 30 argv;
no FABRIC_SUBNET reference remains in scripts/ or docker/.

* Trim the glmv5.1 Dockerfile header to the reviewed comment budget

Removes 14 comment lines of within-file duplication from the header,
bringing the Dockerfile to the 180-line comment target from review:

  - the docker build invocation appeared twice (header and "build
    context" block); keep the first, which also shows DOCKER_IMAGE_NAME
  - WITH_NIXL=0/=1 was explained in the header and again at the ARG;
    keep the ARG-site copy, leave the one-line mention in the connector
    summary
  - BASE_IMAGE override was stated in both STATUS and the closing block;
    fold the ~30-60 min compile time into STATUS and drop the block
  - the "add a vllm_disagg_inference.<model>.Dockerfile per future model"
    guidance duplicates README.MD

Also corrects a false claim inherited from the sibling Dockerfile: the
pin bullet said "validated recipe knobs baked as ENV", which contradicts
section 5 ("The image ships NO runtime recipe / tuning / platform ENV").
The recipe lives in models.yaml and is applied at launch.

Comment-only: the non-comment diff is empty, all 34 Dockerfile
directives are unchanged, and tests/run_all.sh is green (30 argv
assertions + gate checks).

* [GLM-5.1] Bump the component pins to the validated fresh stack

Moves the recipe off the older pin set onto the stack that was
re-validated end-to-end, and records the EP32 fix.

  MoRI    42e895472b08 -> 624002c897a3
  AITER   ROCm/aiter e03fa6040 -> raviguptaamd/aiter 624e43586b
  flydsl  >=0.1.7,<0.1.9 -> ==0.3.1
  vLLM    glm5.1-dsa-wideEP_on_vllm-v0.27 (branch) -> 094820b5d (pinned)
  router  ravgupta/discovery-dp-rank-roundrobin -> ravgupta/dp-roundrobin-on-tip

Why each:

* vLLM 094820b5d is branch glm5.1-dsa-wideEP_on_d626108b: the 10 GLM DSA
  commits rebased onto upstream d626108b (2026-08-20). It includes the MoRI
  combine() original-topk fix (623fdc946b), which resolves the EP32 token
  corruption the header previously called a KNOWN OPEN DEFECT. Measured at
  4P/4D EP32 on the pre-bump stack: NIAH 8k 0/10 -> 10/10, perf-neutral.
  Pinned to a sha rather than a branch so builds are reproducible.

* AITER: newer aiter claims native gfx942 support for gqa64 fp8 decode and
  routes GLM-5.1 to mla_a8w8_qh64_qseqlen1_gqaratio64_v3_ps, which GPU-faults.
  The 7-line fix lets gqa64 fall through to the capture-safe persistent
  view-fold, preserving cudagraph decode (FULL_AND_PIECEWISE) instead of
  forcing eager. Filed upstream as ROCm/aiter#4957. The fork ref is
  TEMPORARY -- revert AITER_REPO to ROCm/aiter once #4957 merges. This is
  what unblocks moving off the old e03fa6040 pin at all.

* MoRI 624002c897a3 and flydsl 0.3.1 are the co-requisites of that aiter
  bump (flydsl_fp8_mqa_logits is missing on 0.1.x).

Validation on the bumped pins: 1P/1D EP8 and 2P/2D EP16, prefill MoRI-HT
eager + decode MoRI-LL FULL_AND_PIECEWISE -- NIAH 2k 10/10, 8k 10/10;
8192/1024 con32 TPOT ~59 ms. 4P/4D EP32 has NOT been re-validated on these
pins (needs an 8-node allocation); the header says so explicitly rather
than implying coverage.

Also refreshes the header comments that described the old pins, so the
rationale text matches the ARGs.

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Fix two claims the pin bump left behind

Follow-up to 094d108, which bumped the pins but left three statements that
no longer match the file.

VLLM_REF is now a 40-char sha, so the REPRODUCIBILITY note contradicted
itself: it still warned that "this default is a BRANCH NAME, so it is
mutable" and then told the reader to pass --build-arg VLLM_REF=<the same
sha> for an auditable rebuild. The mutable-ref hazard is what the bump
removed, so say that instead.

That note also claimed the pin "is the tip every number in models.yaml was
measured on". It is not: the decode-cudagraph table in models.yaml was
measured on d723eb305e and has not been re-measured on 094820b5d. Left as
written it would credit the new stack with the old stack's numbers, which
is the same class of false-provenance claim this PR has already corrected
twice (e8a633d, 59f5fd8).

README.MD's model table still called EP32 "known-broken" while the
Dockerfile header two files away now says it is fixed. Matches the table to
the header's wording, including that EP32 has not been re-validated on the
bumped pins.

Comment- and doc-only: no Dockerfile directive changes, all ARG values
untouched.

* [GLM-5.1] Make 4P/4D EP32 launchable, pin the router, record the validation

Three fixes from validating an image built off this recipe unmodified
(glm5.1-pr206-9f16198, built from PR head 9f16198).

1. EP32 could not boot. GPU_MEMORY_UTILIZATION 0.8 OOMs during EngineCore
   init at 4P/4D ("Tried to allocate 6.00 GiB ... 157.95 GiB allocated"),
   which SIGTERMs every rank; the socket barrier then prints "Waiting for
   nodes" for ~30 min, so it reads as a hang rather than a crash. The MoRI
   dispatch/combine buffers scale with EP width but the fraction did not.
   The default is now derived from decode EP width: 0.70 at EP>=32, 0.8
   below. Measured: 0.70 boots (66.79 GiB KV cache) and holds through
   con=64; 2P/2D EP16 at 0.8 is unchanged (validated at stock in the same
   session). Submit-time -e still overrides.

   This had to go in the launcher, not models.yaml: the -e on the docker run
   line is unconditional, and vllm_disagg.sh skips any yaml key already in
   the environment, so the models.yaml GPU_MEMORY_UTILIZATION key never
   takes effect. Left in place (it documents intent) but the comment now
   says where the live value is. Note this is the only change here outside
   the glmv5.1 Dockerfile.

2. ROUTER_REF was a branch name, so two builds on different days could ship
   different routers - the same mutable-ref hazard the VLLM_REF sha pin
   removed. Pinned to 82dc9811. The comment above it still described the old
   PR ROCm#181 branch and its two commits; upstream has since merged the
   round-robin fix (prefill_dp_round_robin), so only the KV-notify fix is
   carried locally now.

3. STATUS said 4P/4D was fixed but not re-validated on these pins. It is now
   validated, so the claim is replaced with the measurements:

     NIAH, 3 seeds:  2P/2D 2k 9.7/10, 8k 9.7/10 | 4P/4D 2k 9.7/10, 8k 9.3/10
     8192/1024 con32 TPOT median: 2P/2D 60.0 ms | 4P/4D 70.4 ms
     0 failed requests across all 8 cells, 0 GPU faults

   The 9/10s are seed variance, not length collapse - across 12 cells every
   score is 9 or 10 with no pattern by topology or context length - so the
   text says ~93-97% retrieval rather than claiming a clean 10/10. Also
   records that EP32 costs ~17% TPOT over EP16, so nobody reads 4P/4D as a
   latency win. README.MD's model table matched to the same wording.

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Drop the quoted performance figures from the recipe

Comment-only. No ARG, no env value, no code line changes.

Reading this the way an outside reviewer would: the recipe quoted a lot of
specific latency and throughput numbers and none of them said which stack
produced them. That is a problem in both directions.

Some were measured on a stack this recipe no longer pins - the
decode-cudagraph table in models.yaml is from d723eb305e, as 9f16198 already
noted - so a reader would credit the current pins with older results. The
rest were real but were one configuration on one cluster, and a figure like
"TPOT 41.8 ms" sitting in a recipe file reads as a spec that CI or a customer
can hold you to. Neither belongs in a file whose job is to pin components.

Removed the numbers, kept the reasoning that made each setting the right
choice: FULL_AND_PIECEWISE is still justified (faster for decode, gain
narrows as EP widens, costs boot time), max-num-batched-tokens 2048 still
explains the TTFT-vs-latency tradeoff, and the inductor-partition A/B is
still recorded as neutral. Anyone can re-measure; nobody has to trust a
constant that may have gone stale.

The STATUS block keeps the NIAH results - those are a correctness claim and
are what the EP32 fix is actually about - and now states that a latency sweep
was run without quoting it, with a note to re-measure on your own hardware.

Also drops "1P/1D EP8" from the README model table. 1P/1D was validated on an
earlier stack, not on these pins; only 2P/2D EP16 and 4P/4D EP32 were. I
added that claim in error - it is the same false-provenance mistake 9f16198
corrected.

The EP32 OOM comment in the launcher keeps its error string, since that is
what someone chasing the "Waiting for nodes" hang will grep for, but loses
the KV-cache figure.

Co-Authored-By: Claude <noreply@anthropic.com>

* [GLM-5.1] Address Cemberk's 3 MAD-integration design issues

Issue 1 — NIAH is unschedulable and reports nothing:
- Add pyt_vllm_disagg_mori_glm-5.1-fp8_niah card with BENCHMARK_SCRIPT=niah
  and niah tag so it's CI-schedulable and tag-filterable.
- Add --niah mode to parse_to_csv.py: parses NIAH summary lines and emits
  madengine perf.csv rows (retrieval score per context length).
- benchmark_niah.sh now calls parse_to_csv.py after the harness run.

Issues 2 & 3 — GPU_MEMORY_UTILIZATION precedence fix:
- Fix the broken yaml path: slurm now conditionally forwards
  GPU_MEMORY_UTILIZATION only if set by submit-time, and always passes
  _GPU_MEM_UTIL_LAUNCHER_DEFAULT as a separate var.
- vllm_disagg.sh applies the launcher fallback after yaml loads, so:
    submit-time -e  >  models.yaml  >  launcher fallback (0.70@EP>=32, 0.8)
- Remove inert GPU_MEMORY_UTILIZATION from GLM and Qwen yaml (they now
  use the topology-aware fallback).
- Update header comment and README to document the precedence.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: raviguptaamd <ravi.gupta@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ravi Gupta <raviguptaamd@users.noreply.github.com>

Resolves the three conflicts blocking ROCm#244 and folds in one fix that the same
merge makes newly relevant.

Conflicts, all "both sides added here":
  - .gitignore: keep both rule blocks (skill manifest templates from mad-rccl,
    the JAX/MaxText and madengine-artifact rules from develop).
  - scripts/sglang_disagg/models.json: reconciled by unioning the card lists by
    name rather than editing markers, then checked nothing either side added was
    dropped. 21 + 18 over a common 16 -> 23 cards; develop contributes
    pyt_sglang_disagg_mori_io_agentic_qwen3-32b and
    pyt_sglang_disagg_mori_dp_agentic_deepseek-v3.
  - scripts/sglang_disagg/sglang_disagg_mori_io_ep.sh: take develop on both
    hunks. The first is a pure addition (agentic Prometheus metrics). The second
    is develop generalizing the hardcoded benchmark_xPyD.sh into ${_bench_file}
    for the AgentX suite; that variable is defined a few lines up by another,
    non-conflicting develop hunk and defaults to benchmark_xPyD.sh, so taking
    develop's side is backwards compatible.

Verified after resolution: no markers left, the shell script and both JSON files
parse, and both sides' markers survive in the launcher -- mad-rccl's arch gate,
Kimi MLA guard and the three checked/bounded barrier calls, alongside develop's
AGENTIC_SERVER_METRICS and BENCHMARK_SCRIPT_FILE hook.

Also re-gates mori's UMBP unit tests behind BUILD_TESTS in
sglang_disagg_inference_full_overlay (ported from MAD-private
aicomnet_dev_public_gpt_oss, bb7ce03). mori's src/umbp/CMakeLists.txt calls
add_subdirectory(tests) unconditionally; those tests use gtest_discover_tests()
in POST_BUILD mode, which EXECUTES each freshly-linked test binary to enumerate
cases. The binaries link the ROCm/HIP runtime and abort inside `docker build`,
which has no GPU, and the empty output then breaks CMake's string(JSON ...) and
fails the amd_mori wheel build. So ENABLE_MORI_OVERLAY=1 -- the documented knob
for A/B-ing a MoRI commit -- cannot currently build at its own default pin:
verified that the unconditional add_subdirectory(tests) is still present at
MORI_COMMIT=a14e6992 and already fixed on mori main. The patch greps before it
seds, so it self-disables to MORI_UMBP_TESTS_GUARD_SKIPPED once the pin moves
past the upstream fix. Not reproduced end to end here -- only the trigger
condition was confirmed -- and the default path (ENABLE_MORI_OVERLAY=0, MoRI
taken from the sgl-dev base) is unaffected either way.